Augmented Video Prototyping

ABSTRACT

In implementations of augmented video prototyping, a mobile device records augmented video data as a captured video of a recorded scene in an environment, the augmented video data including augmented reality tracking data as 3D spatial information relative to objects in the recorded scene. A video prototyping module localizes the mobile device with reference to the objects in the recorded scene using the 3D spatial information for the mobile device being within boundaries of the recorded scene in the environment. The video prototyping module can generate an avatar for display that represents the mobile device at a current location from a perspective of the recorded scene, and create a spatial layer over a video frame at the current location of the avatar that represents the mobile device. The spatial layer is an interactive interface on which to create an augmented reality feature that displays during playback of the captured video.

BACKGROUND

Generally, augmented reality (AR) applications refer to interactiveinterfaces and techniques used to develop and augment a real-worldenvironment with virtual content, such as with 3D generated graphics,texts, images, and the like onto the physical environment. Designing andprototyping are typical functions of an AR application developmentprocess. The design process generally involves designers,user-experience researchers, clients, and project managers expressingand communicating their design ideas about how an AR application shouldlook and function. However, the prototyping aspect needed to create anAR application generally requires designers with an in-depth knowledgeof advanced programming and 3D modeling skills, which can be overlycumbersome and stifle creativity.

Notably, a designer can incur significant development time and effort todesign and prototype the features and animations of an AR application.Some conventional design applications attempt to simplify some of the ARauthoring and prototyping, such as by abstracting some of theprogramming tasks with simple interactive user interface functions.These abstractions, however, are still programmatic and require an ARdesigner to have some knowledge of advanced programming, as well as 3Dmodeling skills to fully develop a prototype of an AR application. Evenmore limiting, these conventional design applications are designed to beused with 2D inputs (e.g., with a mouse, a keyboard, or touch inputs)rather than using 3D spatial AR interactions that a designer may want toincorporate into an AR application design. Many of the primary authoringand development tasks in the AR design process, such as for 3D objectplacement and manipulation, can be quite tedious when attempting toaccomplish the tasks in only a 2D design environment.

SUMMARY

This Summary introduces features and concepts of augmented videoprototyping, which is further described below in the DetailedDescription and shown in the Figures. This Summary should not beconsidered to describe essential features of the claimed subject matter,nor used to determine or limit the scope of the claimed subject matter.

Aspects of augmented video prototyping are described. Inimplementations, a mobile device includes a camera device that recordsaugmented video data as a captured video of a recorded scene in anenvironment, the augmented video data including augmented reality (AR)tracking data as 3D spatial information relative to objects in therecorded scene. The mobile device implements a video prototyping modulethat can localize the mobile device with reference to the objects in therecorded scene using the 3D spatial information responsive to the mobiledevice being moved within the boundaries of the scene in theenvironment. The video prototyping module can generate an avatar fordisplay over a video frame showing the recorded scene, where the avatarrepresents the mobile device at a current location from the perspectiveof the recorded scene of the environment. The video prototyping modulecan track relative locations in the video frames of the mobile device inthe recorded scene of the environment using the 3D spatial informationas the video frames are displayed, and the avatar is displayable overthe video frames to represent the mobile device at respective locationsin the recorded scene of the environment.

The video prototyping module can also create a spatial layer over thevideo frame at the current location of the avatar that represents themobile device in the recorded scene of the environment, where thespatial layer is an interactive interface on which to create anaugmented reality feature that displays during playback of the capturedvideo. Similar to the avatar that represents the mobile device, thespatial layer can be localized with reference to the objects in thevideo frame of the recorded scene using the 3D spatial information. Inimplementations, an augmented reality feature, such as any type of asketch, image, animation, 3D model, etc. can be associated with thespatial layer at the current location of the spatial layer in the videoframe of the recorded scene. Additionally, the avatar that representsthe mobile device can be linked or associated with the spatial layer,and then as a user moves the mobile device, the spatial layer movesalong a motion path according to the movements of the mobile device fora 3D manipulation of the augmented reality feature that is created onand associated with the spatial layer.

The video prototyping module can record the movements of the spatiallayer as animation actions, and the recorded movements translate to theanimation actions that are applied to the augmented reality feature,which can then be displayed as an animation during playback of thecaptured video. The video prototyping module can map the spatial layertiming and positioning of the augmented reality feature to one or moreof the video frames of the captured video, and the augmented realityfeature displays as the animation during playback of the video frames ofthe captured video.

In other aspects of augmented video prototyping, an augmented realityfeature of a spatial layer can be assigned to an animation feature ofanother spatial layer that is used to guide animation of the augmentedreality feature. As noted above, the spatial layer can be created at thecurrent location of the avatar that represents the mobile device in thevideo frame of the recorded scene. An additional spatial layer can thenbe created over the video frame at a different location of the avatarthat represents the mobile device in the recorded scene. The videoprototyping module can receive an input of a motion path sketch on theadditional spatial layer, and the augmented reality feature of thespatial layer is assigned to the motion path sketch on the additionalspatial layer. The augmented reality feature then displays as ananimation that moves according to the motion path sketch during playbackof the captured video.

BRIEF DESCRIPTION OF THE DRAWINGS

Implementations of augmented video prototyping are described withreference to the following Figures. The same numbers may be usedthroughout to reference similar features and components that are shownin the Figures:

FIG. 1 illustrates an example mobile device in an example environment inwhich aspects of augmented video prototyping can be implemented.

FIGS. 2-4 illustrate examples of recording augmented video data ascaptured video in implementations of augmented video prototyping.

FIG. 5 illustrates an example user interface of a video prototypingmodule which can be used to implement aspects of augmented videoprototyping.

FIGS. 6-11 illustrate an example of creating and adding an augmentedreality feature that is displayable during playback of the capturedvideo in aspects of augmented video prototyping.

FIGS. 12-17 illustrate another example of creating and adding anaugmented reality feature that is displayable with animation duringplayback of the captured video in aspects of augmented videoprototyping.

FIGS. 18-21 illustrates another example of creating and adding anaugmented reality feature that is displayable with animation duringplayback of the captured video in aspects of augmented videoprototyping.

FIGS. 22 and 23 illustrate an animation control feature usable toanimate an augmented reality feature that is displayable with animationduring playback of the captured video in aspects of augmented videoprototyping.

FIG. 24 illustrates another an animation control feature usable toanimate an augmented reality feature that is displayable with animationduring playback of the captured video in aspects of augmented videoprototyping.

FIGS. 25-29 illustrate the playback of the captured video overlaid withthe augmented video prototyping features created as shown and describedabove with reference to FIGS. 6-24 in aspects of augmented videoprototyping.

FIGS. 30-33 illustrate example methods of augmented video prototyping inaccordance with one or more implementations.

FIG. 34 illustrates an example system with an example device that canimplement aspects of augmented video prototyping.

DETAILED DESCRIPTION

Implementations of augmented video prototyping are described, andprovide techniques for rapidly prototyping augmented reality (AR)features and animations over augmented video data of a captured video.The techniques described herein for augmented video prototyping allownon-technical AR designers (e.g., those without advanced programming and3D modeling skills) to quickly develop and animate augmented realitydesign ideas. Generally, the expertise needed as a designer with anin-depth knowledge of advanced programming and 3D modeling skills todesign an AR application can significantly slow the design process.Further, a designer having to think about abstract technical programmingand modeling concepts, rather than about the overall design andanimation features, does not lend to an ideal user experience. Notably,designers and other artistic creatives are more effective when they canwork directly with the design medium, which in this case is in the ARenvironment.

The features of augmented video prototyping described herein enabledesigners to capture video with a camera of a mobile device, such aswith a tablet device or other similar computing device that has acamera, and edit the video directly on the same mobile device thatcaptures the video. The mobile device can include a video prototypingmodule, which implements the features and aspects of augmented videoprototyping as described herein. The video prototyping module includes auser interface that displays on the mobile device, and through which auser of the device can initiate to record and capture the video, as wellas quickly develop and animate augmented reality design ideas that aredisplayable over the captured video as the video is played back forviewing.

Generally, video prototyping is significantly less complicated thanhaving to edit computer code to design and model an AR application.Further, the augmented reality interactions via the user interface ofthe video prototyping module are more intuitive for a designer whodevelops 3D manipulation tasks and animates an augmented realityfeature, rather than trying to configure the same animation tasks in a2D programmatic interface. Notably, the features of augmented videoprototyping combine the benefits of both video prototyping and ARauthoring into a cohesive AR video prototyping system. Further, thefeatures of augmented video prototyping use 3D positioning to createspatial layers over video frames of the captured video, rather thanusing conventional 2D interaction techniques to draw animations over thevideo frames. The techniques described for augmented video prototypinginclude augmented video recording, creating spatial layers over thecaptured video, and an enactment recorder of the video prototypingmodule that allows a user to develop and animate augmented realityfeatures using the same device as was used to capture the video.

As noted above, a user with a mobile device, which also has a camera,can utilize the device to implement the features of augmented videorecording, such as to capture a video recording with synchronized ARdata. The user can initiate use of the camera to record augmented videodata as a captured video of a recorded scene in an environment, such asoutdoors, indoors, in a room, etc. As described in more detail below,individual video frames of the captured video can be sequentiallydisplayed and/or displayed as individual still images on the display ofthe mobile device for augmented video prototyping. The augmented videodata that is recorded as the captured video includes AR tracking data as3D spatial information relative to objects in the recorded scene of theenvironment, as well as 3D spatial information relative to the positionof the camera device (i.e., in the mobile device) while the augmentedvideo data is being recorded. With this unique recording of theaugmented video data and the synchronized AR data, a user of the mobiledevice can develop and animate augmented reality features using bothfamiliar 2D interactions, such as a video frame timeline and sketches,as well as intuitive and spatial AR interactions havingsix-degree-of-freedom manipulations for 3D augmented prototyping.

The techniques for augmented video prototyping include creating spatiallayers over the captured video. A spatial layer allows a designer todevelop and animate augmented reality features that will be displayableover the captured video during playback of the video for viewing. Aspatial layer can display images and graphics on a 3D canvas, and anaugmented reality feature can be composited onto the augmented videorecording similar to placing a 3D object onto a live AR scene. However,in implementations of augmented video prototyping, the design andanimation of the augmented reality feature occurs in an affine manner onthe recorded video feed, transforming and mapping the spatial layer topreserve coordinates, lines, planes, and geometric relations in the 3Dspace. Thus, the designer has more control of the resulting compositionand manipulation of the augmented reality features as 3D objects.

Generally, with the use of spatial layers created and mapped to thecaptured video, being able to create and animate augmented realityfeatures over the video has several unique advantages for a designer.These advantages include ideation, which provides that the video can befrozen to display individual video frames on which to create an ARexperience, allowing the designer to explore and iterate different ideason the same experience as an essential aspect of the design process.Another advantage is being able to provide an intuitive understanding ofdepth perception when developing and animating an augmented realityfeature over the captured video. For designers using conventional designtechniques, knowing how far a virtual object (e.g., an augmented realityfeature) is relative to the AR device is a common problem with creatingand developing animated AR features. However, with augmented videoprototyping, the designer can intentionally capture a video withmultiple complementary angles to view and adjust the rendering of theaugmented reality feature as a virtual object with respect to thecaptured scene. Additionally, from the standpoint of collaboration,video is a natural format for collaborative review and feedback, and thedesigner can quickly export the video and send it to anothercollaborator for input and feedback on an AR prototype.

In implementations of augmented video prototyping, the enactmentrecorder allows a designer to create and develop animation over thecaptured video. Notably, the animation of an augmented reality featurecan be created by manipulating a spatial layer directly, such as bychanging its transformation and appearance, or indirectly through asketch-based interface. For direct manipulation of a spatial layer, theuser of the mobile device can “grab” the spatial layer using the mobiledevice and move it around in the recorded scene of the environment. Thetransformation of the movement in 3D space (as six-degrees-of-freedommovement) can be recorded as a motion path, which allows the designer toexpress complex motion trajectory in a more intuitive manner than with atraditional, spline-based interface. Through the user interface of thevideo prototyping module, the user can also adjust the size andappearance of a spatial layer, such as the opacity of the layer and acropping mask. For an indirect manipulation of the spatial layer, theuser of the mobile device can assign a virtual object (e.g., anaugmented reality feature) to a pre-defined motion path to define theanimation of the augmented reality feature. A user can create thepre-defined motion path on another spatial layer either by creating anew sketch on the spatial layer or by importing a motion path, forexample from an image file.

These animation recordings can then be mapped to the augmented videoutilizing the AR tracking data and a timeline of the video frames tocreate the animation of an augmented reality feature in the video space.The features of augmented video prototyping allow a designer to mockupan AR experience by creating and animating an augmented reality featurethat can then be mapped to individual video frames of the capturedvideo, in a sense, “freezing” the video and developing layeredinteractions over the video to prototype augmented video in an ARenvironment

While features and concepts of augmented video prototyping can beimplemented in any number of different devices, systems, networks,environments, and/or configurations, implementations of augmented videoprototyping are described in the context of the following exampledevices, systems, and methods.

FIG. 1 illustrates an example environment 100 in which aspects ofaugmented video prototyping can be implemented. The example environment100 includes a mobile device 102, which implements features of a videoprototyping module 104 for augmented video prototyping, as describedherein. The mobile device 102 can be implemented as any type ofcomputing device, client device, mobile phone, tablet device,communication, entertainment, gaming, media playback, and/or other typeof electronic and/or computing device. In this example environment 100,the mobile device 102 may include any number and combination ofdifferent components as further described with reference to the exampledevice shown in FIG. 33. For example, the mobile device 102 includes amemory 106 and a processor 108, as well as a display device 110 and acamera device 112 used to record video.

The mobile device 102 implements the video prototyping module 104, whichmay be implemented as a module that includes independent processing,memory, and/or logic components functioning as a computing and/orelectronic device integrated with the mobile device 102. Alternativelyor in addition, the video prototyping module 104 can be implemented insoftware, in hardware, or as a combination of software and hardwarecomponents. In this example, the video prototyping module 104 isimplemented as a software application or modules, such as executablesoftware instructions (e.g., computer-executable instructions) that areexecutable with a processing system (e.g., the processor 108) of themobile device 102 to implement the described techniques of augmentedvideo prototyping. As a software application or module, the videoprototyping module 104 can be stored in memory of the device (e.g., inthe device memory 106), or in any other suitable memory device orelectronic data storage implemented with the video prototyping module.Alternatively or in addition, the video prototyping module 104 may beimplemented in firmware and/or at least partially in computer hardware.For example, at least part of the video prototyping module 104 may beexecutable by a computer processor, and/or at least part of the videoprototyping module may be implemented in logic circuitry.

The camera device 112 of the mobile device 102 can be used to recordaugmented video data 114 as a captured video 116 of a scene 118 in theexample environment 100. For example, the mobile device 102 is shown asa tablet device with a camera used to record the video of the scene 118,which is a room with several objects 120, such as a table, a stool,couches, etc., as well as a television or other similar type of mediadisplay device 122. The augmented video data 114 is recorded by thecamera device 112 and can be stored in the device memory 106, as well asdisplayed as the captured video 116 on the display device 110 of themobile device. Notably, as described in more detail below, individualvideo frames 124 of the captured video 116 can be sequentially displayedand/or displayed as individual still images on the display device 110 ofthe mobile device for augmented video prototyping.

In implementations, the augmented video data 114 includes augmentedreality (AR) tracking data 126 as 3D spatial information relative to theobjects 120 in the captured video 116 of the recorded scene 118, as wellas relative to the position of the camera device 112 (i.e., in themobile device 102) while the augmented video data 114 is being recorded.The 3D spatial information can be captured along with the augmentedvideo data 114 from a changing user viewpoint, to encompass differentperspectives of the scene 118 in the environment 100. The mobile device102 can be enabled with an augmented reality framework, and thesynchronized AR tracking data 126 can be obtained from already developeddata tracking systems and/or determined from the video processingpipeline, such as based on optical flow and surface reconstruction. Themobile device 102 includes device sensors 128, such as may beimplemented as components of an inertial measurement unit (IMU). Thedevice sensors 128 can be implemented with various sensors, such as agyroscope, an accelerometer, and/or other types of motion and locationsensors to sense motion of the device at a particular location. Thedevice sensors 128 can generate sensor data vectors havingthree-dimensional parameters (e.g., rotational vectors in x, y, andz-axis coordinates) indicating location, position, acceleration,rotational speed, and/or orientation of the device.

Generally, the device sensors 128 can include hardware sensors thatgather data used by the video prototyping module 104 to establish adevice origin 130 when the mobile device 102 is used to record theaugmented video data 114 as the captured video 116 of the recorded scene118 in the example environment 100. As a user moves the mobile device102 within the boundaries of the captured video 116 of the recordedscene 118, the video prototyping module 104 can use the 3D spatialinformation of the AR tracking data 126 to localize the mobile device102 with reference to the objects 120 in a video frame 124 of therecorded scene, and determine current and subsequent device locations132 within the boundaries of the scene 118 of the environment.

FIGS. 2-4 further illustrate the camera device 112 of the mobile device102 being used to record the augmented video data 114 as the capturedvideo 116 of the recorded scene 118 in the example environment 100. Forexample, as shown in FIG. 2, a user who is holding the mobile device 102and recording the video moves the mobile device into the scene 118 ofthe environment, generally toward the table. A camera view 200 of thevideo that is being recorded by the camera device 112 is also shown forperspective, and shows what the user who is holding the mobile device102 can see while recording the video. Although the user in theenvironment is blocking a view of the table (e.g., one of the objects120 in the scene 118), the camera view 200 shows that the table, as wellas the other objects in the scene, are being recorded as the capturedvideo 116.

Similarly, as shown in FIG. 3, the user who is holding the mobile device102 and recording the video is shown sitting on a couch within the scene118 of the environment 100. In this example, the user is adding agesture 300 that will later correspond to an animation feature added tothe video for augmented video prototyping. As shown in a camera view 302of the video that is being recorded by the camera device 112, the userrecords a swipe-up gesture 304 across the media display device 122 aspart of the recording of the captured video 116. Next, as shown in FIG.4, the user continues recording the video and adds another gesture 400that will also correspond to another animation feature added to thevideo for augmented video prototyping. As shown in a camera view 402 ofthe video that is being recorded by the camera device 112, the userrecords a movement gesture 404 towards the media display device 122 aspart of the recording of the captured video 116.

Returning to the discussion of FIG. 1, the video prototyping module 104implemented by the mobile device 102 includes a user interface 134, viawhich the user can interact with the device for augmented videoprototyping of the captured video 116. As illustrated and describedabove with reference to FIGS. 2-4, the captured video is recorded as theaugmented video data 114 of the recorded scene 118 in the exampleenvironment 100. The augmented video data 114 includes the AR trackingdata 126 that provides 3D spatial information of the position of thecamera device 112 (i.e., in the mobile device 102) relative to theobjects 120 in the captured video 116 of the recorded scene 118. Thevideo prototyping module 104 can also generate an avatar 136 as arepresentation of the mobile device 102, which can be displayed in therecorded scene of the captured video. The avatar 136 is a visualizationof the current position of the mobile device 102 in the environment 100as the user moves the mobile device around in the environment after thecaptured video 116 is recorded.

FIG. 5 illustrates an example 500 of the user interface 134 for thevideo prototyping module 104, via which the user of the mobile device102 can interact with the video frames 124 of the captured video 116 foraugmented video prototyping. In this example, the user interface 134 isshown in a playback view, which includes a playback timeline 502 with aframe locator 504 that indicates the portion of the captured video 116being displayed in the playback view as one of the video frames 124. Asnoted above, the individual video frames 124 of the captured video 116can be sequentially displayed with a play-pause control 506 and/ordisplayed as individual still images utilizing a stop control 508 on thedisplay device 110 of the mobile device 102 in the playback view of theuser interface 134 for augmented video prototyping.

The user interface 134 in this example 500 also includes a sketch-enactcontrol 510 that the user can select to create a spatial layer in avideo frame 124 of the captured video 116 of the recorded scene 118 inthe environment. As described in more detail below, a representation ofthe mobile device 102 can be displayed as the avatar 136 in the recordedscene of the captured video, and a spatial layer can be created at theposition of the device representation in a video frame of the video. Theavatar 136 is a visualization of the current position of the mobiledevice 102 in the environment 100 as the user moves the mobile devicearound in the environment. The user interface 134 includes a hide-showcontrol 512 that the user can toggle to either display the avatar 136 asthe visual representation of the mobile device in the environment, orhide the display of the avatar.

The user interface 134 also includes a grab-drop control 514 that theuser can select to initiate moving a spatial layer that has been createdcorresponding to the user moving the mobile device 102 in theenvironment. The user interface 134 has an undo selector 516, as well asa color selector 518 that initiates a color selection palette whenselected by the user. The user interface 134 also includes animationcontrol features 520 that are selectable by the user of the mobiledevice 102 to animate when a spatial layer appears and disappears, aswell as orientation and deletion of the layer. Each of these userinterface controls are described in more detail below with reference tothe examples of augmented video prototyping shown and described in FIGS.6-29.

Returning to the discussion of FIG. 1, the video prototyping module 104implemented by the mobile device 102 can be used to generate a spatiallayer 138 at the current location of the avatar 136 that represents themobile device 102 in a video frame 124 of the recorded scene 118. Aspatial layer 138 can display images and graphics on a 3D canvas (e.g.,similar to a canvas in Photoshop™), and is rendered on 3D planeprimitives, where an augmented reality feature can be composited ontothe augmented video recording similarly to placing a 3D object onto alive AR scene. A spatial layer can also define the position andorientation of a 3D model. For example, 3D text can be placed in spaceon the spatial layer, or a dimensional sign could be positioned so thatits front is aligned with the spatial layer. However, in implementationsof augmented video prototyping, the design and animation of theaugmented reality feature 140 occurs in an affine manner on the recordedvideo feed, transforming and mapping the spatial layer 138 to preservecoordinates, lines, planes, and geometric relations in the 3D space.

As further described below with reference to the examples shown in thefigures, the spatial layer 138 is an interactive interface on which tocreate an augmented reality feature 140 that displays during playback ofthe captured video 116. Similar to the avatar 136 that represents themobile device, a spatial layer 138 can be localized with reference tothe objects 120 in the video frame 124 of the recorded scene 118 usingthe 3D spatial information in the AR tracking data 126. Generally, thespatial layer 138 can be created and a mapping 142 is used to localizethe spatial layer in the video frame, and subsequent video frames 124,as the position of the spatial layer changes relative to the videoframes during playback of the captured video 116. Notably, a spatiallayer 138 can be used to develop, create, insert, and/or animate anytype of media, such as sketches, images, 3D models, animated models, andthe like. Additionally, the augmented reality feature 140 as any type ofthe media can be associated with the spatial layer by the videoprototyping module.

The video prototyping module 104 also includes an enactment recorder144, which can be utilized by the user of the device to initiateanimation actions 146 of an augmented reality feature 140 that has beencreated on a spatial layer 138. The enactment recorder 144 can beinitiated from the user interface 134 by user selection of thesketch-enact control 510. The enactment recorder 144 can be implementedso that the user of the mobile device 102 can create and developanimation over the captured video 116. Notably, the animation of anaugmented reality feature 140 can be created by manipulating a spatiallayer 138 directly, such as by changing its transformation andappearance, or indirectly through a sketch-based interface. For directmanipulation of a spatial layer 138, the user of the mobile device 102can “grab” (or attach) the spatial layer using the mobile device andmoving it around in the recorded scene 118 of the environment 100. Thetransformation of the movement in 3D space (as six-degrees-of-freedommovement) can be recorded as an animation action 146 (e.g., a motionpath), which allows the user to develop complex motion trajectory in anintuitive manner.

Through the user interface 134 of the video prototyping module 104, theuser can also adjust the size and appearance of a spatial layer 138,such as the opacity of the layer and a cropping mask. For an indirectmanipulation of the spatial layer 138, the user of the mobile device 102can assign an augmented reality feature 140 to a pre-defined motion pathto define the animation of the augmented reality feature. A user cancreate the pre-defined motion path on another spatial layer either bycreating a new sketch on the spatial layer or by importing a motionpath, for example from an image file. The recordings of the animationactions 146 can then be mapped to the augmented video data 114 utilizingthe AR tracking data 126 and a timeline of the video frames 124 tocreate the animation of an augmented reality feature 140 in the videospace. The features of augmented video prototyping allow a designer tomockup an AR experience by creating and animating an augmented realityfeature 140 that can then be mapped to individual video frames 124 ofthe captured video 116.

FIGS. 6-11 illustrate the mobile device 102 being used to add anaugmented reality feature that is displayable during playback of thecaptured video 116. For example, as shown in FIG. 6, a user of themobile device 102 moves the device into the scene 118 of the environment100 and holds the device on the table (e.g., one of the objects 120 inthe environment). The video prototyping module 104 can localize themobile device 102 with reference to the objects 120 in the video frame124 of the recorded scene using the 3D spatial information of the ARtracking data 126 responsive to the mobile device being moved within theboundaries of the scene 118 in the environment. This reflects the 3Dexperience by changing the camera position with respect to the timelineof the video frames 124 of the video. In this example, the video frame124 of the scene generally corresponds to the camera view 200 of thevideo that is being recorded by the user with the camera device 112 ofthe scene shown in FIG. 2, with a closer view of the table. A designview 600 of the user interface 134 of the video prototyping module 104is also shown for perspective in FIG. 6, and shows the user interfaceover a video frame 124 of the captured video 116, which is what the userwho is holding the mobile device 102 sees on the device.

The design view 600 also displays the avatar 136 that is generated bythe video prototyping module 104 for display over the video frame 124 ofthe recorded scene 118 in the environment 100. As shown in this example,the avatar 136 represents the mobile device 102 at the current locationof the device in the recorded scene of the captured video 116, which inthis instance, is on the table. The video prototyping module 104 cantrack the relative locations of the mobile device 102 in the videoframes 124 of the recorded scene 118 in the environment 100 using the 3Dspatial information of the AR tracking data 126 as the video frames aredisplayed, and the avatar 136 is displayable over the video frames torepresent the mobile device at locations in the respective video framesof the captured video.

The user of the mobile device 102 can then initiate creating a spatiallayer 138 over the video frame 124 at the current location of the avatar136 that represents the mobile device 102 in the recorded scene 118 ofthe environment. As shown in FIG. 7 in a design view 700 of the userinterface 134 of the video prototyping module 104, the user interface134 includes the sketch-enact control 510 that the user can select tocreate the spatial layer 138 over the video frame 124 of the capturedvideo 116 of the recorded scene 118 in the environment. The videoprototyping module 104 can create the spatial layer 138 over the videoframe at the current location of the avatar 136 (i.e., on the table inFIG. 6) that represents the mobile device in the scene 118 of theenvironment 100.

The spatial layer 138 is an interactive interface on which to create anaugmented reality feature that displays during playback of the capturedvideo. Similar to the avatar 136 that represents the mobile device 102,the spatial layer 138 can be localized with reference to the objects ofthe scene shown in the video frame 124 using the 3D spatial informationof the AR tracking data 126. As further shown in FIG. 7, the user canpick up and move the mobile device 102 for augmented video prototypingafter initiating the spatial layer 138 being created to appear on thetable in the video frame. Notably, the relative position of the avatar136 has moved in the design view 700, corresponding to the currentlocation of the mobile device 102 in the scene of the environment, wherethe user has picked up and moved the mobile device.

FIG. 8 illustrates an augmented reality feature 800 being sketched onthe user interface 134 of the video prototyping module 104, as alsoshown in a design view 802, which shows what the user who is holding themobile device 102 sees on the device. For example, the user who issitting in the environment 100 can initiate the sketch mode of the videoprototyping module 104, such as by user selection of the sketch-enactcontrol 510 in the user interface 134. The user of the mobile device 102can then sketch the augmented reality feature 800 on the spatial layer138 shown on the display of the device. In this example, the user of themobile device is prototyping the augmented reality feature 800 as asimple display screen that displays the current temperature and anindication of the weather.

FIG. 9 further illustrates a color selection feature of the userinterface 134 for the video prototyping module 104. The user who issitting in the environment 100 and sketching the augmented realityfeature 800 on the spatial layer 138, as further shown in a design view900, can select the color selector control 518 to initiate a drop-downdisplay of a color selection palette 902 from which the user can selectand/or change sketch colors. Although shown in grayscale in the figure,the color selection palette 902 is representative of a full range ofcolors that can be selected and used in the sketch of the augmentedreality feature.

FIG. 10 further illustrates the augmented reality feature 800 beingsketched on the user interface 134 of the video prototyping module 104,as also shown in a design view 1000, which shows what the user who isholding the mobile device 102 sees on the device. After selecting orchanging sketch colors using the color selection palette 902, the userin this example adds the sketch feature 1002 to the augmented realityfeature 800, as shown in the design view 1000.

FIG. 11 illustrates an example 1100 of the completed prototype for theaugmented reality feature 800, as shown in the user interface 134 of thevideo prototyping module 104 over the video frame 124 of the capturedvideo 116. In this example, the augmented reality feature 800 is shownover the video frame at the position of the spatial layer 138, which inthis instance, is on the table of the recorded scene captured in thevideo. During playback of the video frames 124 of the captured video116, the augmented reality feature 800 will appear over the video duringthe closer views of the table, as previously recorded by the user withthe camera device 112 of the mobile device 102. While FIGS. 8-10illustrate the user creating the augmented reality feature usingsketching, the video prototyping module 104 could also create theaugmented reality feature by placing an image, a video, a 3D model, ananimation, etc. on a spatial layer.

FIGS. 12-17 illustrate the mobile device 102 being used to add anotheraugmented reality feature that is displayable with animation duringplayback of the captured video 116. For example, as shown in FIG. 12,the user of the mobile device 102 moves the device within the scene 118of the environment 100 and approaches the media display device 122. Thevideo prototyping module 104 can localize the mobile device 102 withreference to the media display device 122 in the video frame 124 of therecorded scene using the 3D spatial information of the AR tracking data126 responsive to the mobile device being moved within the boundaries ofthe scene 118 in the environment. In this example, the video frame 124of the recorded scene generally corresponds to the camera view 302 ofthe video that is being recorded by the user with the camera device 112of the scene shown in FIG. 3, with the user adding the swipe-up gesture304 across the media display device 122. A design view 1200 of the userinterface 134 of the video prototyping module 104 is also shown forperspective in FIG. 12, and shows the user interface over a video frame124 of the captured video 116, which is what the user who is holding themobile device 102 sees on the device.

The design view 1200 also displays the avatar 136 that is generated bythe video prototyping module 104 for display over the video frame 124 ofthe recorded scene 118 in the environment 100. As shown in this example,the avatar 136 represents the mobile device 102 at the current locationof the device in the recorded scene of the captured video 116, which inthis instance, is approaching the media display device 122. The user ofthe mobile device 102 can then initiate to create a spatial layer 138over the video frame 124 at the current location of the avatar 136 thatrepresents the mobile device 102 in the scene 118 of the environment.

As shown in FIG. 13 in a design view 1300 of the user interface 134 ofthe video prototyping module 104, the user interface 134 includes thesketch-enact control 510 that the user can select to create the spatiallayer 138 over the video frame 124 of the captured video 116 of therecorded scene 118 in the environment. The video prototyping module 104can create the spatial layer 138 over the video frame at the currentlocation of the avatar 136 (i.e., on the media display device 122) thatrepresents the mobile device in the scene 118 of the environment 100. Asnoted above, the spatial layer 138 is an interactive interface on whichto create an augmented reality feature that displays during playback ofthe captured video. As further shown in FIG. 14, the user can move themobile device 102 for augmented video prototyping after initiating thespatial layer 138 being created to appear on the media display device122. FIG. 14 also illustrates that the user can manipulate the size andshape of the spatial layer 138 on the user interface 134 of the videoprototyping module 104, as also shown in a design view 1400, which showswhat the user who is holding the mobile device 102 sees on the device.

FIG. 15 illustrates an augmented reality feature 1500 being added as agraphic or image on the user interface 134 of the video prototypingmodule 104, as also shown in a design view 1502, which shows what theuser who is holding the mobile device 102 sees on the device. Forexample, the user who is sitting in the environment 100 can insert thegraphic or image via the user interface 134 of the video prototypingmodule 104. The user of the mobile device 102 can select and add theimage as the augmented reality feature 1500 on the spatial layer 138shown on the display of the device. In this example, the user of themobile device is prototyping the augmented reality feature 1500 toappear as if the user turns on the media display device 122 to displaythe augmented reality feature 1500.

This correlates with FIG. 16 in which the user of the mobile device 102can initiate the enactment recorder 144 of the video prototyping module104 via the sketch-enact control 510 on the user interface 134 to createan animation action 146 of the spatial layer 138, as shown in the designview 1600 on the mobile device. Notably, when the user's hand appears inthe video frame 124 at the first position 1602, near the bottom of themedia display device 122, the spatial layer 138 is collapsed orminimized so as not to display the augmented reality feature 1500 (e.g.,the image of the cat). Transitioning to FIG. 17, and as shown in thedesign view 1700, the user of the mobile device 102 continues theanimation action 146 of the spatial layer 138 over subsequent videoframes 124. Notably, as the user's hand transitions through the swipe-upgesture 304 across the media display device 122 to the second position1702, near the top of the media display device 122, the spatial layer138 is expanded or maximized to again display the augmented realityfeature 1500. During playback of the video frames 124 of the capturedvideo 116, the augmented reality feature 1500 will appear over the videoand correspond to the swipe-up gesture 304 across the media displaydevice 122, giving the appearance of the user turning on the mediadisplay device to show the image of the cat.

FIGS. 18-21 illustrate the mobile device 102 being used to add anotheraugmented reality feature that is displayable with animation duringplayback of the captured video 116. For example, as shown in FIG. 18,the user of the mobile device 102 is using the device within the scene118 of the environment 100, and holds up the device to add an augmentedreality feature that corresponds to the movement gesture 404 towards themedia display device 122. In this example, the video frame 124 of thescene generally corresponds to the camera view 402 of the video that isbeing recorded by the user with the camera device 112 of the scene shownin FIG. 4, with the user adding the movement gesture 404 towards themedia display device 122. A design view 1800 of the user interface 134of the video prototyping module 104 is also shown for perspective inFIG. 18, and shows the user interface over a video frame 124 of thecaptured video 116, which is what the user who is holding the mobiledevice 102 sees on the device.

The design view 1800 displays the avatar 136 that is generated by thevideo prototyping module 104 for display over the video frame 124 of therecorded scene 118 in the environment 100. As shown in this example, theavatar 136 represents the mobile device 102 at the current location ofthe device in the recorded scene of the captured video 116. The user ofthe mobile device 102 can then initiate to create a spatial layer 138over the video frame 124 at the current location of the avatar 136 thatrepresents the mobile device 102 in the scene 118 of the environment.The user interface 134 includes the sketch-enact control 510 that theuser can select to create the spatial layer 138 over the video frame 124of the captured video 116 of the recorded scene 118 in the environment.As described above, the video prototyping module 104 can create thespatial layer 138 over the video frame at the current location of theavatar 136 that represents the mobile device in the scene 118 of theenvironment 100.

As further shown in FIG. 19, the user can move the mobile device 102 foraugmented video prototyping after initiating the spatial layer 138 beingcreated to appear at the position in front of the media display device122, and over the user's hand as shown in the captured video. The usercan also manipulate the size and shape of the spatial layer 138 on theuser interface 134 of the video prototyping module 104, as also shown ina design view 1900, which shows what the user who is holding the mobiledevice 102 sees on the device. Note in the design view 1900 that theuser has toggled the hide-show control 512 on the user interface 134 tohide the display of the avatar 136, which was shown in the design view1800 above in FIG. 18. FIG. 19 illustrates the augmented reality feature1902 being added as a sketch on the user interface 134 of the videoprototyping module 104, as also shown in the design view 1900, whichshows what the user who is holding the mobile device 102 sees on thedevice. For example, the user who is sitting in the environment 100 caninitiate the sketch mode of the video prototyping module 104, such as byuser selection of the sketch-enact control 510 in the user interface134. The user of the mobile device 102 can then sketch the augmentedreality feature 1902 on the spatial layer 138 shown on the display ofthe device.

FIGS. 20 and 21 illustrate an animation control feature usable toanimate an augmented reality feature in aspects of augmented videoprototyping as described herein. In implementations, the avatar 136 thatrepresents the mobile device 102 can be linked or associated with aspatial layer 138, and then as the user moves the mobile device withinthe environment 100, the spatial layer 138 moves along a motion pathaccording to the movements of the mobile device for a 3D manipulation ofan augmented reality feature 140 that is created on and/or associatedwith the spatial layer. For example, FIG. 20 illustrates the user of themobile device 102 moving the device within the scene 118 of theenvironment 100 as the user approaches the media display device 122. Asdescribed above, the video prototyping module 104 can localize themobile device 102 with reference to objects and features shown in thevideo frame 124 of the recorded scene using the 3D spatial informationof the AR tracking data 126 responsive to the mobile device being movedwithin the boundaries of the scene 118 in the environment. A design view2000 of the user interface 134 of the video prototyping module 104 isalso shown for perspective, and shows the user interface over a videoframe 124 of the captured video 116, which is what the user who isholding the mobile device 102 sees on the device.

The design view 2000 also displays the avatar 136 that is generated bythe video prototyping module 104 for display over the video frame 124 ofthe recorded scene 118 in the environment 100. As shown in this example,the avatar 136 represents the mobile device 102 at the current locationof the device in the scene of the captured video 116, which in thisinstance, is approaching the media display device 122. The user of themobile device 102 can then select the grab-drop control 514 on the userinterface 134 to link or associate the avatar 136 with the spatial layer138. Notably, this technique links the avatar 136 to the augmentedreality feature 1902, and the user can then initiate moving the spatiallayer 138 (and the augmented reality feature 1902) corresponding tomovements of the mobile device 102 in the environment. In this example,the user of the mobile device is prototyping the augmented realityfeature 1902 to appear as if the user flings a sketch of a heart towardsthe image of the cat shown on the media display device 122.

For example, as shown in FIG. 21, and in a design view 2100, the spatiallayer 138, along with the augmented reality feature 1902, moves as ananimation action 2102 according to movements 2104 of the mobile device102 as the user moves the device in the environment 100. The videoprototyping module 104 can record the movements of the spatial layer 138as the animation actions 146, and the recorded movements translate tothe animation action 2102 that is applied to the augmented realityfeature 1902, which can then be displayed as an animation duringplayback of the captured video 116. The video prototyping module 104 canalso map the spatial layer timing and positioning as the mapping 142 ofthe augmented reality feature 1902 to one or more of the video frames124 of the captured video 116 during the animation sequence, and theaugmented reality feature 1902 (i.e., the sketch of the heart) displaysas an animation during playback of the corresponding video frames of thecaptured video 116.

FIGS. 22 and 23 further illustrate the animation control feature usableto animate an augmented reality feature in aspects of augmented videoprototyping, as described above with reference to FIGS. 21 and 22.Notably, an avatar 136 that represents the mobile device 102 can belinked or associated with a spatial layer 138, and then as the usermoves the mobile device within the scene 118 of the environment, thespatial layer 138 moves along with the movement of the mobile device fora 3D manipulation of an augmented reality feature 140 that is created onand/or associated with the spatial layer. For example, FIG. 22illustrates the user of the mobile device 102 standing within the scene118 of the environment near the media display device 122. As describedabove, the video prototyping module 104 can localize the mobile device102 with reference to objects and features shown in the video frame 124of the recorded scene using the 3D spatial information of the ARtracking data 126 responsive to the mobile device being moved within theboundaries of the scene 118 in the environment. A design view 2200 ofthe user interface 134 of the video prototyping module 104 is also shownfor perspective, and shows the user interface over a video frame 124 ofthe captured video 116, which is what the user who is holding the mobiledevice 102 sees on the device.

The design view 2200 also displays the avatar 136 that is generated bythe video prototyping module 104 for display over the video frame 124 ofthe recorded scene 118 in the environment 100. As shown in this example,the avatar 136 represents the mobile device 102 at the current locationof the device in the recorded scene of the captured video 116, which inthis instance, is proximate the media display device 122. The user ofthe mobile device 102 can then select the grab-drop control 514 on theuser interface 134 to link or associate the avatar 136 with the spatiallayer 138. Notably, this technique links the avatar 136 to the augmentedreality feature 1500 (i.e., the image of the cat), and the user can theninitiate moving the spatial layer 138 along with the avatar 136 in adirection 2202 corresponding to a movement 2204 of the mobile device 102in the environment. In this example, the user of the mobile device isprototyping the augmented reality feature 1500 to appear as if the catshakes back and forth after the user flings the heart towards the cat,as shown and described above with reference to FIGS. 18-21.

FIG. 23 further shows, in a design view 2300, that the spatial layer138, along with the augmented reality feature 1500, moves as ananimation action 2302 corresponding to a movement 2304 of the mobiledevice 102 as the user moves the device in the environment 100. Thevideo prototyping module 104 can record the movements of the spatiallayer 138 as the animation actions 146, and the recorded movementstranslate to the back and forth animation actions that are applied tothe augmented reality feature 1500, which can then be displayed as ananimation during playback of the captured video 116. The videoprototyping module 104 can also map the spatial layer timing andpositioning as the mapping 142 of the augmented reality feature 1500 toone or more of the video frames 124 of the captured video 116 during theanimation sequence, and the augmented reality feature 1500 (i.e., theimage of the cat) displays as an animation during playback of thecorresponding video frames of the captured video 116.

FIG. 24 illustrates another animation control feature of augmented videoprototyping as described herein. In implementations, an augmentedreality feature 140 of a spatial layer 138 can be assigned to ananimation feature of another spatial layer that is used to guideanimation of the augmented reality feature. For example, a design view2400 of the user interface 134 of the video prototyping module 104 isshown for perspective, and shows the spatial layer 138 and theassociated augmented reality feature 1500 (i.e., the image of the cat),which is what the user who is holding the mobile device 102 sees on thedevice. The user interface 134 includes the sketch-enact control 510that the user can select to create an additional spatial layer 2402 overthe video frame 124 of the captured video 116 of the recorded scene 118in the environment.

The video prototyping module 104 can receive an input of a motion pathsketch 2404 on the additional spatial layer 2402, as drawn by the userof the mobile device 102. The augmented reality feature 1500 of thespatial layer 138 can then be assigned to the motion path sketch 2404 onthe additional spatial layer, and the augmented reality feature 1500then displays as an animation that moves according to the motion pathsketch 2404 during playback of the corresponding video frames of thecaptured video 116. The enactment of the additional spatial layer 2402is assigned to the selected spatial layer 138 to animate the augmentedreality feature 1500. In this example, the image of the cat will appearto come out of the media display device 122, moving left and downwardoff of the display screen of the mobile device. Notably, this assignmentfeature of augmented video prototyping can be used to animate multipleaugmented reality features at the same time, such as to animateraindrops. For example, hundreds of spatial layers 138 that each includea raindrop as an augmented reality feature 140 can be assigned to thesame additional spatial layer with a line-sketch that animates thedropping motion of all of the raindrops together.

FIGS. 25-29 illustrate the playback of the captured video 116 overlaidwith the augmented video prototyping features created as shown anddescribed above with reference to FIGS. 6-24. For example, as thecaptured video 116 is played back for viewing, FIG. 25 illustrates anexample 2500 of the augmented reality feature 800 being shown as asimple display screen that displays the current temperature and anindication of the weather on the table in the scene 118 as captured inthe video. During playback of the captured video 116, the augmentedreality feature 800 will appear over the video during the closer viewsof the table, as previously recorded by the user with the camera device112 of the mobile device 102. This prototype of the augmented realityfeature 800 generally corresponds to the camera view 200 of the videothat is being recorded by the user as shown in FIG. 2, with a closerview of the table, and corresponds to prototyping of the augmentedreality feature 800 as shown and described with reference to FIGS. 6-11.

FIG. 26 further illustrates an example 2600 of the animation of theaugmented reality feature 1500 (i.e., the image of the cat) shown as theuser swipe-up gesture 304 across the media display device 122, givingthe appearance of the user turning on the media display device 122 todisplay the image of the cat. As shown in the video playback of thecaptured video at 2602, the user's hand appears in the video at a firstposition 2604 near the bottom of the media display device 122, andtransitioning to the video playback at 2606, the user swipes-up acrossthe media display device 122 to a second position 2608 of the user'shand near the top of the display, giving the appearance of turning onthe device to display the image of the cat. This prototype of theaugmented reality feature 1500 generally corresponds to the camera view300 of the video that is being recorded by the user as shown in FIG. 3,and corresponds to the prototyping of the augmented reality feature 1500as shown and described with reference to FIGS. 12-17.

FIG. 27 further illustrates an example 2700 of the animation of theaugmented reality feature 1902 (i.e., the sketch of the heart) shown asthe movement gesture 404 towards the media display device 122, givingthe appearance of the user flinging a heart towards the cat shown on thedisplay. As shown in the video playback of the captured video at 2702,the user's hand appears in the video at a first position 2704, andtransitioning to the video playback at 2706, the user's hand movestowards the media display device 122 as the movement gesture to a secondposition 2708 of the user's hand. The animation of the augmented realityfeature 1902 follows the user's hand as it appears in the video at thefirst position 2704 and moves towards the media display device 122 asthe movement gesture to the second position 2708 of the user's hand.Transitioning to the video playback 2710, the animation of the augmentedreality feature 1902 continues, leaving the user's hand and flinging theheart to the cat on the media display device. This prototype of theaugmented reality feature 1902 generally corresponds to the camera view400 of the video that is being recorded by the user as shown in FIG. 4,and corresponds to the prototyping of the augmented reality feature 1902as shown and described with reference to FIGS. 18-21.

FIG. 28 further illustrates an example 2800 of the animation of theaugmented reality feature 1500 (i.e., the image of the cat), giving theappearance of the cat shaking back and forth after the user flings theheart towards the cat. As shown in the video playback 2802 of thecaptured video, the augmented reality feature 1500 moves left over themedia display device 122, transitions to the video playback 2804 andmoves right over the media display device, and further transitions tothe video playback 2806 and moves back left over the media displaydevice in the captured video. This prototype of the augmented realityfeature 1500 generally corresponds to the prototyping of the augmentedreality feature as shown and described with reference to FIGS. 22 and23.

FIG. 29 further illustrates an example 2900 of another animation of theaugmented reality feature 1500 (i.e., the image of the cat), giving theappearance of the cat coming out of the media display device 122, movingleft and downward off of the display screen of the mobile device 102. Asshown in the video playback 2902 of the captured video, the augmentedreality feature 1500 begins to move across the media display device 122,transitions to the video playback 2904 and continues to move out of themedia display device, and further transitions to the video playback 2906and continues to move off of the display screen of the mobile device.This prototype of the augmented reality feature 1500 generallycorresponds to the prototyping of the augmented reality feature as shownand described with reference to FIG. 24.

Example methods 3000, 3100, 3200, and 3300 are described with referenceto respective FIGS. 30, 31, 32, and 33 in accordance with one or moreaspects of augmented video prototyping. Generally, any of thecomponents, modules, methods, and operations described herein can beimplemented using software, firmware, hardware (e.g., fixed logiccircuitry), manual processing, or any combination thereof. Someoperations of the example methods may be described in the generalcontext of executable instructions stored on computer-readable storagememory that is local and/or remote to a computer processing system, andimplementations can include software applications, programs, functions,and the like. Alternatively, or in addition, any of the functionalitydescribed herein can be performed, at least in part, by one or morehardware logic components, such as, and without limitation,Field-programmable Gate Arrays (FPGAs), Application-specific IntegratedCircuits (ASICs), Application-specific Standard Products (ASSPs),System-on-a-chip systems (SoCs), Complex Programmable Logic Devices(CPLDs), and the like.

FIG. 30 illustrates example method(s) 3000 for augmented videoprototyping, and is generally described with reference to the videoprototyping module implemented by a computing device as shown anddescribed with reference to FIGS. 1-29. The order in which the method isdescribed is not intended to be construed as a limitation, and anynumber or combination of the method operations can be combined in anyorder to implement a method, or an alternate method.

At 3002, augmented video data is recorded as a captured video of arecorded scene in an environment, the augmented video data includingaugmented reality (AR) tracking data as 3D spatial information relativeto objects in the recorded scene. For example, the mobile device 102includes the camera device 112 that can be used to record the augmentedvideo data 114 as the captured video 116 of the recorded scene 118 inthe example environment 100. The augmented video data 114 includes theaugmented reality (AR) tracking data 126 as 3D spatial informationrelative to the objects 120 in the captured video 116 of the recordedscene 118, as well as relative to the position of the camera device 112(i.e., in the mobile device 102) while the augmented video data 114 isbeing recorded.

At 3004, video frames of the captured video showing the recorded scenein the environment are displayed. For example, the mobile device 102includes the display device 110 to display the video frames 124 of thecaptured video 116 showing the recorded scene 118 in the environment100.

At 3006, the mobile device is localized with reference to the objects inthe recorded scene of the environment using the 3D spatial informationresponsive to the mobile device being within boundaries of the scene inthe environment. For example, the mobile device 102 may move to withinthe boundaries of the scene 118 in the environment 100, and the videoprototyping module 104 implemented by the mobile device can use the 3Dspatial information of the AR tracking data 126 to localize the mobiledevice 102 with reference to the objects 120 in a video frame 124 of therecorded scene, as well as determine current and subsequent devicelocations 132 within the boundaries of the scene 118 of the environment.

At 3008, an avatar is generated for display over a video frame showingthe recorded scene, the avatar representing the mobile device at acurrent location from the perspective of the recorded scene of theenvironment. For example, the video prototyping module 104 implementedby the mobile device 102 can generate the avatar 136 as a representationof the mobile device 102, which can be displayed in the recorded sceneof the captured video. The avatar 136 is a visualization of the currentposition of the mobile device 102 in the environment 100 as the usermoves the mobile device around in the environment after the capturedvideo 116 is recorded.

FIG. 31 illustrates example method(s) 3100 for augmented videoprototyping, and is generally described with reference to the videoprototyping module implemented by a computing device as shown anddescribed with reference to FIGS. 1-29. The order in which the method isdescribed is not intended to be construed as a limitation, and anynumber or combination of the method operations can be combined in anyorder to implement a method, or an alternate method.

At 3102, augmented video data is recorded as a captured video of arecorded scene in an environment, the augmented video data includingaugmented reality (AR) tracking data as 3D spatial information relativeto objects in the recorded scene. For example, the mobile device 102includes the camera device 112 that can be used to record the augmentedvideo data 114 as the captured video 116 of the recorded scene 118 inthe example environment 100. The augmented video data 114 includes theaugmented reality (AR) tracking data 126 as 3D spatial informationrelative to the objects 120 in the captured video 116 of the recordedscene 118, as well as relative to the position of the camera device 112(i.e., in the mobile device 102) while the augmented video data 114 isbeing recorded.

At 3104, video frames of the captured video showing the recorded scenein the environment are displayed. For example, the mobile device 102includes the display device 110 to display the video frames 124 of thecaptured video 116 showing the recorded scene 118 in the environment100.

At 3106, a representation of the mobile device that captured theaugmented video data is generate for display over a video frame, and isshown at a current location of the mobile device in the recorded sceneof the environment. For example, the video prototyping module 104implemented by the mobile device 102 can generate the representation ofthe mobile device 102 as an avatar 136 that is displayed in the videoframes 124 of the recorded scene 118 at the current location of themobile device. The avatar 136 is a visualization of the current positionof the mobile device 102 in the environment 100 as the user moves themobile device around in the environment after the captured video 116 isrecorded.

At 3108, a spatial layer is created at the current location of therepresentation of the mobile device in the video frame of the recordedscene. For example, the video prototyping module 104 implemented by themobile device 102 can create the spatial layer 138 at the currentlocation of the avatar 136 that represents the mobile device 102 in avideo frame 124 of the recorded scene 118, and the spatial layer islocalized with reference to the objects in the video frame of therecorded scene using the 3D spatial information. The spatial layer 138is an interactive interface on which to create an augmented realityfeature 140 that displays during playback of the captured video 116.

At 3110, an augmented reality feature is associated with the spatiallayer at the current location in the video frame of the recorded scene.For example, the video prototyping module 104 implemented by the mobiledevice 102 can associate an augmented reality feature 140, such as anytype of a sketch, image, animation, 3D model, etc. with the spatiallayer 138 that is created at the current location of the avatar 136 thatrepresents the mobile device 102 in a video frame 124 of the recordedscene 118.

FIG. 32 illustrates example method(s) 3200 for augmented videoprototyping, and is generally described with reference to the videoprototyping module implemented by a computing device as shown anddescribed with reference to FIGS. 1-29. The order in which the method isdescribed is not intended to be construed as a limitation, and anynumber or combination of the method operations can be combined in anyorder to implement a method, or an alternate method.

At 3202, augmented video data is recorded as a captured video of arecorded scene in an environment, the augmented video data includingaugmented reality (AR) tracking data as 3D spatial information relativeto objects in the recorded scene. For example, the mobile device 102includes the camera device 112 that can be used to record the augmentedvideo data 114 as the captured video 116 of the recorded scene 118 inthe example environment 100. The augmented video data 114 includes theaugmented reality (AR) tracking data 126 as 3D spatial informationrelative to the objects 120 in the captured video 116 of the recordedscene 118, as well as relative to the position of the camera device 112(i.e., in the mobile device 102) while the augmented video data 114 isbeing recorded.

At 3204, relative locations of the mobile device are tracked in thevideo frames of the recorded scene in the environment using the 3Dspatial information as the video frames are displayed. For example, thevideo prototyping module 104 implemented by the mobile device 102 cantrack relative locations of the mobile device in the video frames 124 ofthe recorded scene 118 in the environment 100 using the 3D spatialinformation as the video frames are displayed.

At 3206, an avatar is displayed in the video frames to represent themobile device at locations in the respective video frames of therecorded scene. For example, the video prototyping module 104implemented by the mobile device 102 generates the avatar 136 that isdisplayed in the video frames 124 of the recorded scene 118 as arepresentation of the mobile device 102. The avatar 136 is avisualization of the current position of the mobile device 102 in theenvironment 100 as the user moves the mobile device around in theenvironment after the captured video 116 is recorded.

At 3208, a spatial layer is created at the current location of theavatar that represents the mobile device in the video frame of therecorded scene, the spatial layer being localized with reference to theobjects in the video frame of the recorded scene using the 3D spatialinformation. For example, the video prototyping module 104 implementedby the mobile device 102 can create the spatial layer 138 at the currentlocation of the avatar 136 that represents the mobile device 102 in avideo frame 124 of the recorded scene 118. The spatial layer 138 is aninteractive interface on which to create an augmented reality feature140 that displays during playback of the captured video 116, and aspatial layer 138 can be localized with reference to the objects 120 inthe video frame 124 of the recorded scene 118 using the 3D spatialinformation in the AR tracking data 126.

At 3210, an augmented reality feature is associated with the spatiallayer at the current location in the video frame of the recorded scene.For example, the video prototyping module 104 implemented by the mobiledevice 102 can associate an augmented reality feature 140, such as anytype of a sketch, image, animation, 3D model, etc. with the spatiallayer 138 that is created at the current location of the avatar 136 thatrepresents the mobile device 102 in a video frame 124 of the recordedscene 118.

At 3212, the avatar that represents the mobile device is associated withthe spatial layer. For example, the video prototyping module 104implemented by the mobile device 102 can link or associate the avatar136 with the spatial layer 138, which links the avatar 136 to theaugmented reality feature 140 of the spatial layer, and the user canthen initiate moving the spatial layer 138 (and the augmented realityfeature) corresponding to movements of the mobile device 102 in theenvironment.

At 3214, animation actions are recorded as movements of the spatiallayer. For example, the video prototyping module 104 implemented by themobile device 102 can initiate to record movements of the spatial layer138 as the animation actions 146 according to movements of the mobiledevice 102 as the user moves the device in the environment 100. Therecorded movements translate to animation actions 146 that can beapplied to an augmented reality feature 140 that is associated with thespatial layer, and the animation actions can then be displayed as ananimation during playback of the captured video 116.

At 3216, the spatial layer is mapped to one or more of the video framesof the captured video to create spatial layer timing and positioning ofthe augmented reality feature that will display during playback of thecaptured video. At 3218, the augmented reality feature is displayed asan animation that moves according to the animation actions applied tothe augmented reality feature and/or the 3D manipulation during playbackof the captured video. For example, the video prototyping module 104implemented by the mobile device 102 can map the spatial layer timingand positioning as the mapping 142 of the augmented reality feature 140to one or more of the video frames 124 of the captured video 116 duringthe animation sequence, and the augmented reality feature 140 displaysas an animation during playback of the corresponding video frames 124 ofthe captured video 116. Notably, the augmented reality feature 140 isdisplayed as an animation that moves according to the animation actions146 applied to the augmented reality feature and/or the 3D manipulationsduring playback of the captured video.

FIG. 33 illustrates example method(s) 3300 for augmented videoprototyping, and is generally described with reference to the videoprototyping module implemented by a computing device as shown anddescribed with reference to FIGS. 1-29. The order in which the method isdescribed is not intended to be construed as a limitation, and anynumber or combination of the method operations can be combined in anyorder to implement a method, or an alternate method.

At 3302, augmented video data is recorded as a captured video of arecorded scene in an environment, the augmented video data includingaugmented reality (AR) tracking data as 3D spatial information relativeto objects in the recorded scene. For example, the mobile device 102includes the camera device 112 that can be used to record the augmentedvideo data 114 as the captured video 116 of the recorded scene 118 inthe example environment 100. The augmented video data 114 includes theaugmented reality (AR) tracking data 126 as 3D spatial informationrelative to the objects 120 in the captured video 116 of the recordedscene 118, as well as relative to the position of the camera device 112(i.e., in the mobile device 102) while the augmented video data 114 isbeing recorded.

At 3304, an avatar is generated for display over a video frame showingthe recorded scene, the avatar representing the mobile device at acurrent location from the perspective of the recorded scene of theenvironment. For example, the video prototyping module 104 implementedby the mobile device 102 can generate the avatar 136 as a representationof the mobile device 102, which can be displayed in the recorded sceneof the captured video. The avatar 136 is a visualization of the currentposition of the mobile device 102 in the environment 100 as the usermoves the mobile device around in the environment after the capturedvideo 116 is recorded.

At 3306, a spatial layer is created at the current location of theavatar that represents the mobile device in the video frame of therecorded scene, the spatial layer localized with reference to theobjects in the video frame of the recorded scene using the 3D spatialinformation. For example, the video prototyping module 104 implementedby the mobile device 102 can create the spatial layer 138 at the currentlocation of the avatar 136 that represents the mobile device 102 in avideo frame 124 of the recorded scene 118. The spatial layer 138 is aninteractive interface on which to create an augmented reality feature140 that displays during playback of the captured video 116, and aspatial layer 138 can be localized with reference to the objects 120 inthe video frame 124 of the recorded scene 118 using the 3D spatialinformation in the AR tracking data 126.

At 3308, an augmented reality feature is associated with the spatiallayer at the current location in the video frame of the recorded scene.For example, the video prototyping module 104 implemented by the mobiledevice 102 can associate an augmented reality feature 140, such as anytype of a sketch, image, animation, 3D model, etc. with the spatiallayer 138 that is created at the current location of the avatar 136 thatrepresents the mobile device 102 in a video frame 124 of the recordedscene 118.

At 3310, an additional spatial layer is created over the video frame ata different location of the avatar that represents the mobile device inthe recorded scene. For example, the video prototyping module 104implemented by the mobile device 102 can create the additional spatiallayer 2402 at a different location of the avatar 136 that represents themobile device 102 in the recorded scene.

At 3312, an input of a motion path sketch is received on the additionalspatial layer, and at 3314, an augmented reality feature of the spatiallayer is assigned to the motion path sketch on the additional spatiallayer. For example, the video prototyping module 104 implemented by themobile device 102 can receive an input of the motion path sketch 2404 onthe additional spatial layer 2402, as drawn by the user of the mobiledevice 102, and the augmented reality feature 140 of the spatial layer138 can then be assigned to the motion path sketch 2404 on theadditional spatial layer 2402.

At 3316, the augmented reality feature is displayed as an animation thatmoves according to the motion path sketch during playback of thecaptured video. For example, the video prototyping module 104implemented by the mobile device 102 initiates the augmented realityfeature 140 to display as an animation that moves according to themotion path sketch 2404 during playback of the corresponding videoframes 124 of the captured video 116. The enactment of the additionalspatial layer 2402 is assigned to the selected spatial layer 138 toanimate the augmented reality feature 140.

FIG. 34 illustrates an example system 3400 that includes an exampledevice 3402, which can implement techniques of augmented videoprototyping. The example device 3402 can be implemented as any of thecomputing devices, mobile devices, server devices, and/or servicesdescribed with reference to the previous FIGS. 1-33, such as any type ofcomputing device, client device, mobile phone, tablet, communication,entertainment, gaming, media playback, and/or other type of device. Forexample, the mobile device 102 may be implemented as the example device3402.

The example device 3402 includes communication devices 3404 that enablewired and/or wireless communication of device data 3406, such as any ofthe augmented video data, AR tracking data, video frames, sensor data,as well as computer applications data and content that is transferredfrom one computing device to another, and/or synched between multiplecomputing devices. The device data 3406 can include any type of audio,video, image, and/or graphic data that is generated by applicationsexecuting on the device. The communication devices 3404 can also includetransceivers for cellular phone communication and/or for network datacommunication.

The device 3402 also includes input/output (I/O) interfaces 3408, suchas data network interfaces that provide connection and/or communicationlinks between the device, data networks, and other devices. The I/Ointerfaces can be used to couple the device to any type of components,peripherals, and/or accessory devices, such as a digital camera device3410 and/or computer input device that may be integrated with theexample device 3402. The I/O interfaces also include data input portsvia which any type of data, media content, and/or inputs can bereceived, such as user inputs to the device, as well as any type ofaudio, video, and/or image data received from any content and/or datasource.

The device 3402 includes a processor system 3412 that may be implementedat least partially in hardware, such as with any type ofmicroprocessors, controllers, and the like that process executableinstructions. The processor system 3412 can include components of anintegrated circuit, programmable logic device, a logic device formedusing one or more semiconductors, and other implementations in siliconand/or hardware, such as a processor and memory system implemented as asystem-on-chip (SoC). Alternatively or in addition, the device 3402 canbe implemented with any one or combination of software, hardware,firmware, or fixed logic circuitry that may be implemented withprocessing and control circuits. The device 3402 may further include anytype of a system bus or other data and command transfer system thatcouples the various components within the device. A system bus caninclude any one or combination of different bus structures andarchitectures, as well as control and data lines.

The example device 3402 also includes computer-readable storage memory3414, such as data storage devices implemented in hardware that can beaccessed by a computing device, and that provide persistent storage ofdata and executable instructions (e.g., software applications, modules,programs, functions, and the like). The computer-readable storage memorydescribed herein excludes propagating signals. Examples ofcomputer-readable storage memory include volatile memory andnon-volatile memory, fixed and removable media devices, and any suitablememory device or electronic data storage that maintains data forcomputing device access. The computer-readable storage memory 3414 caninclude various implementations of random access memory (RAM), read-onlymemory (ROM), flash memory, and other types of storage memory in variousmemory device configurations.

The computer-readable storage memory 3414 provides storage of the devicedata 3406 and various device applications 3416, such as an operatingsystem that is maintained as a software application with thecomputer-readable storage memory and executed by the processor system3412. In this example, the device 3402 includes a video prototypingmodule 3418 that implements the described techniques of augmented videoprototyping, and may be implemented with hardware components and/or insoftware as one of the device applications 3416, such as when theexample device 3402 is implemented as the mobile device 102 shown anddescribed with reference to FIGS. 1-33. An example of the videoprototyping module 3418 includes the video prototyping module 104 thatis implemented as a software application and/or as hardware componentsby the mobile device 102 as described with reference to FIGS. 1-33. Inimplementations, the video prototyping module 3418 may includeindependent processing, memory, and logic components as a computingand/or electronic device integrated with the device 3402.

The device 3402 also includes an audio and/or video system 3420 thatgenerates audio data for an audio device 3422 and/or generates displaydata for a display device 3424. The audio device and/or the displaydevice include any devices that process, display, and/or otherwiserender audio, video, display, and/or image data. In implementations, theaudio device and/or the display device are integrated components of theexample device 3402. Alternatively, the audio device and/or the displaydevice are external, peripheral components to the example device. Inembodiments, at least part of the techniques described for augmentedvideo prototyping may be implemented in a distributed system, such asover a “cloud” 3426 in a platform 3428. The cloud 3426 includes and/oris representative of the platform 3428 for services 3430 and/orresources 3432.

The platform 3428 abstracts underlying functionality of hardware, suchas server devices (e.g., included in the services 3430) and/or softwareresources (e.g., included as the resources 3432), and connects theexample device 3402 with other devices, servers, etc. The resources 3432may also include applications and/or data that can be utilized whilecomputer processing is executed on servers that are remote from theexample device 3402. Additionally, the services 3430 and/or theresources 3432 may facilitate subscriber network services, such as overthe Internet, a cellular network, or Wi-Fi network. The platform 3428may also serve to abstract and scale resources to service a demand forthe resources 3432 that are implemented via the platform, such as in aninterconnected device embodiment with functionality distributedthroughout the system 3400. For example, the functionality may beimplemented in part at the example device 3402 as well as via theplatform 3428 that abstracts the functionality of the cloud 3426.

Although implementations of augmented video prototyping have beendescribed in language specific to features and/or methods, the appendedclaims are not necessarily limited to the specific features or methodsdescribed. Rather, the specific features and methods are disclosed asexample implementations of augmented video prototyping, and otherequivalent features and methods are intended to be within the scope ofthe appended claims. Further, various different examples are describedand it is to be appreciated that each described example can beimplemented independently or in connection with one or more otherdescribed examples.

1. A mobile device implemented for augmented video prototyping, themobile device comprising: a camera device to record augmented video dataas a captured video of a recorded scene in an environment, the augmentedvideo data including augmented reality (AR) tracking data as 3D spatialinformation relative to objects in the recorded scene; a display deviceto display video frames of the captured video showing the recorded scenein the environment; a video prototyping module implemented at leastpartially in computer hardware, the video prototyping module configuredto: localize the mobile device with reference to the objects in therecorded scene of the environment using the 3D spatial informationresponsive to the mobile device being within boundaries of the recordedscene in the environment; and generate an avatar for display over avideo frame showing the recorded scene, the avatar representing themobile device at a current location from a perspective of the recordedscene of the environment.
 2. The mobile device as recited in claim 1,wherein the video prototyping module is configured to create a spatiallayer over the video frame at the current location of the avatar thatrepresents the mobile device in the recorded scene of the environment,the spatial layer localized with reference to the objects in therecorded scene using the 3D spatial information.
 3. The mobile device asrecited in claim 2, wherein the spatial layer is an interactiveinterface on which to create an augmented reality feature that displaysduring playback of the captured video, the augmented reality featureincluding at least one of a sketch, an image, an animation, or a 3Dmodel.
 4. The mobile device as recited in claim 3, wherein the videoprototyping module is configured to record animation actions asmovements of the spatial layer, and the animation actions are applied tothe augmented reality feature that displays as an animation duringplayback of the captured video.
 5. The mobile device as recited in claim3, wherein the spatial layer is mapped to one or more of the videoframes of the captured video, and the augmented reality feature displaysas an animation during playback of the captured video.
 6. The mobiledevice as recited in claim 3, wherein the video prototyping module isconfigured to create spatial layer timing and positioning of theaugmented reality feature that displays during playback of the capturedvideo.
 7. The mobile device as recited in claim 3, wherein the videoprototyping module is configured to: associate the avatar thatrepresents the mobile device with the spatial layer; move the spatiallayer along a motion path according to movements of the mobile devicefor a 3D manipulation of the augmented reality feature; and theaugmented reality feature displays as an animation that moves accordingto the 3D manipulation during playback of the captured video.
 8. Themobile device as recited in claim 2, wherein the video prototypingmodule is configured to: create an additional spatial layer over thevideo frame at a different location of the avatar that represents themobile device in the recorded scene; receive an input of a motion pathsketch on the additional spatial layer; assign an augmented realityfeature of the spatial layer to the motion path sketch on the additionalspatial layer; and the augmented reality feature displays as ananimation that moves according to the motion path sketch during playbackof the captured video.
 9. The mobile device as recited in claim 1,wherein the video prototyping module is configured to track relativelocations in the video frames of the mobile device in the recorded sceneof the environment using the 3D spatial information as the video framesare displayed, and the avatar is displayable over the video frames torepresent the mobile device at respective locations in the recordedscene of the environment.
 10. In a digital medium environment foraugmented video prototyping, a method implemented by a mobile device,the method comprising: recording augmented video data as a capturedvideo of a recorded scene in an environment, the augmented video dataincluding augmented reality (AR) tracking data as 3D spatial informationrelative to objects in the recorded scene; displaying video frames ofthe captured video showing the recorded scene in the environment;localizing the mobile device with reference to the objects in therecorded scene of the environment using the 3D spatial informationresponsive to the mobile device being within boundaries of the recordedscene in the environment; and generating an avatar for display over avideo frame showing the recorded scene, the avatar representing themobile device at a current location from a perspective of the recordedscene of the environment.
 11. The method as recited in claim 10, furthercomprising: creating a spatial layer over the video frame at the currentlocation of the avatar that represents the mobile device in the recordedscene of the environment, the spatial layer localized with reference tothe objects in the recorded scene using the 3D spatial information. 12.The method as recited in claim 11, wherein the spatial layer is aninteractive interface on which to create an augmented reality featurethat displays during playback of the captured video.
 13. The method asrecited in claim 12, further comprising: recording animation actions asmovements of the spatial layer, and the animation actions applied to theaugmented reality feature that displays as an animation during playbackof the captured video.
 14. The method as recited in claim 12, furthercomprising: mapping the spatial layer to one or more of the video framesof the captured video, and the augmented reality feature displaying asan animation during playback of the captured video.
 15. The method asrecited in claim 12, further comprising: creating spatial layer timingand positioning of the augmented reality feature that displays duringplayback of the captured video.
 16. The method as recited in claim 12,further comprising: associating the avatar that represents the mobiledevice with the spatial layer; moving the spatial layer along a motionpath according to movements of the mobile device for a 3D manipulationof the augmented reality feature; and displaying the augmented realityfeature as an animation that moves according to the 3D manipulationduring playback of the captured video.
 17. The method as recited inclaim 12, further comprising: creating an additional spatial layer overthe video frame at a different location of the avatar that representsthe mobile device in the recorded scene; receiving an input of a motionpath sketch on the additional spatial layer; assigning an augmentedreality feature of the spatial layer to the motion path sketch on theadditional spatial layer; and displaying the augmented reality featureas an animation that moves according to the motion path sketch duringplayback of the captured video.
 18. The method as recited in claim 12,further comprising: tracking relative locations in the video frames ofthe mobile device in the recorded scene of the environment using the 3Dspatial information as the video frames are displayed; and displayingthe avatar over the video frames to represent the mobile device atrespective locations in the recorded scene of the environment.
 19. Amobile device implemented for augmented video prototyping, the mobiledevice comprising: a camera device to record augmented video data as acaptured video of a recorded scene in an environment, the augmentedvideo data including augmented reality (AR) tracking data as 3D spatialinformation relative to objects in the scene; a video prototyping moduleimplemented at least partially in computer hardware, the videoprototyping module configured to: localize the mobile device withreference to the objects in the recorded scene of the environment usingthe 3D spatial information responsive to the mobile device being withinboundaries of the recorded scene in the environment; and create aspatial layer over a video frame at a current location of an avatar thatrepresents the mobile device in the recorded scene of the environment,the spatial layer localized with reference to the objects in therecorded scene and the spatial layer is an interactive interface onwhich to create an augmented reality feature that displays duringplayback of the captured video.
 20. The mobile device as recited inclaim 19, wherein the video prototyping module is configured to: recordanimation actions as movements of the spatial layer, the animationactions applied to the augmented reality feature; map timing andpositioning of the spatial layer to one or more of the video frames ofthe captured video; and the augmented reality feature is displayable asan animation during playback of the captured video.